Correct marshalling of remotely-mapped data #4191
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
otoolep
force-pushed
the
full_marshal
branch
from
e54bd04 to
2262055
Compare
otoolep
changed the title
otoolep
force-pushed
the
full_marshal
branch
2 times, most recently
from
9ed59fb to
720d368
Compare
otoolep
force-pushed
the
full_marshal
branch
2 times, most recently
from
644ae43 to
f85b162
Compare
otoolep
force-pushed
the
full_marshal
branch
from
f85b162 to
e43ad09
Compare
| func (lm *SelectMapper) Open() error { | ||
| // If in aggregate mode, initialize the map-and-reduce functions. Both local and | ||
| // remote mappers need this. | ||
| if !lm.rawMode { |
There was a problem hiding this comment.
This was moved up here because it now needs to be done before the remote mapper is opened.
|
+1 |
[ci skip]
|
Comment corrected @corylanou Merging now. |
otoolep
added a commit
that referenced
this pull request
Sep 22, 2015
Correct marshalling of remotely-mapped data
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fundamental problem was over-use of
interface{}which was preventing proper unmarshalling. This change instead makes use of[]byteand[][]bytewhere necessary, so proper unmarshalling takes place on the local side. This is how the code operated before the DQ-refactor, but this bug was not caught since only some aggregate functions were affected by the bug.countandsumworked fine in DQ mode, butmean(for example) did not.The other key change is making the correct unmarshalling functions available to the remote mapper code. Without this change, a fix was impossible since the remote mapper wouldn't know the correct types, e.g.
meanMapOutputfor the unmarshalling of aggregate data.This solution is not ideal, but more changes are due in this area very soon, so this patch aims to work with the existing framework.
I tested this by setting a single-node into forced-remote mapping mode and ran
meanqueries. Before this fix, the system panicked with the trace in #4170. With the fix in place, everything works fine.Fixes #4170